Intuitive computing methods and systems

ABSTRACT

The present disclosure relates generally to mobile devices and content recognition. One claim recites a method employing a mobile device having a processor configured to perform one or more acts of the method, the mobile device including at least one sensor. The method includes: obtaining information from the sensor; and selecting a user profile from among a plurality of different user profiles based on the information. Other claims and combinations are provided as well.

RELATED APPLICATION DATA

This application is a continuation of U.S. patent application Ser. No.13/614,724, filed Sep. 13, 2012, which claims the benefit of U.S.Provisional Patent Application No. 61/535,299, filed Sep. 15, 2011. Thisapplication is generally related to U.S. patent application Ser. No.12/797,503, filed Jun. 9, 2010, published as US 2011-0161076 A1. Thedisclosures of the above patent documents are each incorporated hereinby reference, in their entireties.

TECHNICAL FIELD

The present specification concerns a variety of technologies; someconcern enabling mobile devices (e.g., smart phones and other devices)to identify content and make intelligent inferences regarding the same.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a block diagram corresponding to a mobile device.

FIG. 2A shows a mobile device display screen held in a first position.

FIG. 2B shows a mobile device display screen held in a second position.

FIG. 2C shows a mobile device display screen held in a third position.

FIG. 3 shows a mobile device display screen displaying a time indicatorgraphic.

DETAILED DESCRIPTION

Cell phones have evolved from single purpose communication tools, tomulti-function computer platforms. “There's an app for that” is afamiliar refrain.

Over six hundred thousand applications are available for smartphones—offering an overwhelming variety of services. However, each ofthese services must be expressly identified and launched by the user.This is a far cry from the vision of ubiquitous computing, dating backover twenty years, in which computers demand less of our attention,rather than more. A truly “smart” phone would be one that takesactions—autonomously—to fulfill inferred or anticipated user desires.

One company, bump Technologies (referred to here as “Bump,” with acapital B), seeks to use mobile phones to exchange contact data andother information. In Bump's model, two mobiles physically (or simulate)bump each other to initiate the exchange. Generally, there are two partsto the bump process: the app running on two mobile devices and amatching algorithm running on the Bump servers in the cloud. The appuses the phone's sensors to literally “feel” the bump, and it sends thatinformation to the Bump servers. The matching algorithm listens to thebumps from phones around the world and pairs up phones that felt thesame bump. Then Bump routes information between the two phones in eachpair. Bump uses various techniques to limit the pool of potentialmatches, including location information and characteristics of the bumpevent. If bumping occurs in a particularly dense area (e.g., at aconference), and it cannot resolve a unique match after a single bump,Bump may ask for an additional bump. Bump uses location information asone of the ways to limit the number of other phones checked to determinethe correct match. Basically, if a phone is in Chicago, such locationinfo is used so that bump need not be compared with bumps coming in fromJapan, Europe, New York, etc. For this reason, Bump may require thatlocation services be turned on and that users authorize the use of theirlocation information. If a user wants to restrict information, they can“un-check” information on a user interface. This limits the informationthat is transferred via the bump.

Digimarc Corp., the assignee of the present application, owns U.S. Pat.No. 7,123,740, which is incorporated herein by reference, in itsentirety. The '740 patent describes how a PDA screen can show an imagesuch as a picture of the PDA's owner. The picture can be digitallywatermarked to serve as a business card. This picture can be beamed toan associate's device (e.g., PDA) at conferences, for example. Theassociate images the picture (or digitally submits to a watermarkreader) and links directly to picture provider's contact page or similardestination for sales, etc.

I have improvements and alternative implementations to transfer contactand other information and data, including providing different userprofiles according to environment or need. For example, transfer ofdifferent user profiles can be streamlined, perhaps with selection of adifferent profile being unnoticed to another party.

A user profile preferably includes information associated with a user,such as name, contact information, links, etc. The format of a userprofile (e.g., HTLM, XML, vCard, etc.) is less important than theunderlining information. And, a user will benefit from multipledifferent user profiles. For example, I may have a first user profilefor use when transferring contact information to a friend. This “frienduser profile” may include more personal information, information I wishto be somewhat restricted like home phone and address, personal emailaccount, family pictures, IP address to my family blog, social mediaaccount information (e.g., Facebook, Twitter, etc.), etc. I also mayhave a business profile, a more professional slice of information likeprofessional credentials, Linked-in account information, work phone andemail, company information, etc. And, I may have a retail profile, whichcould include information to facilitate a transaction like credit cardinformation, bank account, shipping address, preferred online retailer,shipping method preference, etc. (Of course there could be a number ofdifferent profiles, each including different or overlappinginformation.)

A user can enter information for different profiles, e.g., via an apprunning on a mobile device or via an internet website. The userpreferably designates or names each user profile for use in a particularoccasion (e.g., a “friend user profile” or a “business user profile,”etc.). Profiles can be indexed or associated with a content identifier.For example, a first profile may be associated with a first contentidentifier and a second profile may be associated with a second contentidentifier, and so on.

One way to control transfer of different user profiles utilizesdifferent content identifiers provided in different situations. Take,for example, two mobile devices: a first device and a second device.FIG. 1 shows an exemplary mobile device including, e.g., elements suchas a microphone, an optical sensor (e.g., an image and/or video camera),a processor, a display/touchscreen, a physical user interface, a RFtransceiver, location module (e.g., GPS), network adaptor and memory.The memory may store operating system software, user interface software(e.g., an “app” to manage and transfer user profiles), contentrecognition software, other functional software modules, etc. Of course,mobile devices may include more or less features and will also benefitfrom the present disclosure.

Both mobile devices can be running an app (or other softwareapplication) to help control the exchange of user profiles. The app mayprovide a user interface(s) to help guide the exchange.

The first device wishes to transfer information (e.g., a user profile)to the second device. The first mobile device's display screen mayinclude a background image or other graphic. The second device includesan optical sensor, e.g., an image and/or video camera. The seconddevice's camera captures optical data representing the first device'sdisplay. This action initiates transfer of a user profile, e.g., myfriend user profile.

The second device may be equipped with a content recognizer (e.g., theapp may include such or call such). The content recognizer generates acontent identifier for the image content (e.g., from the background orgraphic displayed on the first mobile device's display). This contentidentifier can be forwarded to a remote database, e.g., hosted in thecloud, in which is stored previously populated user profiles. In oneimplementation, the content identifier is used to index a database tofind particular user profile associated with the first mobile device orwith a user of the first mobile device. A corresponding user profile canbe forwarded to the requesting mobile device, or can be associated(e.g., in the cloud) with an account associated with the second mobiledevice.

Content identification can take different forms. In one form, digitalwatermarking is used. The first device's background or graphic mayinclude a digital watermark hidden therein. A watermark may include aplurality of components. For example, one component may carry a message(or payload), while another component may serve to identify theorientation of the message in a combined signal. The orientationcomponent may provide information such as rotation, scale andtranslation of the watermark message. Typically, a watermark detectorlooks for a watermark signal in a potentially distorted version (e.g.,skewed or disoriented with respect to an original orientation) of thecombined watermark signal, and computes its orientation, sometimesrelative to the expected original orientation of the watermark signal.

A watermark payload can be decoded by the second device and forwarded tothe remote database. The payload can be used to locate a user profile.(In another embodiment, the second device forwards image data, perhapsfiltered, to the remote database or other remote location. The imagedata is processed remotely from the second mobile device to decode thedigital watermark.) Technology for encoding/decoding watermarks isdetailed, e.g., in Digimarc's U.S. Pat. Nos: 6,614,914 and 6,122,403; inNielsen's U.S. Pat. Nos. 6,968,564 and 7,006,555; and in Arbitron's U.S.Pat. Nos: 5,450,490, 5,764,763, 6,862,355, and 6,845,360. Thedisclosures of these patents are incorporated herein by reference, intheir entireties, along with Digimarc's U.S. Pat. Nos. 7,958,359 and6,947,571.

Another form of content identification utilizes so-called“fingerprinting.” Fingerprinting derives or extracts a unique identifierfrom a content item itself. For example, perceptual features arestatistically or mathematically characterized, and suchcharacterizations are used as a content identifier. Once a contentidentifier has been computed it can be used by the second mobile deviceto obtain a profile. Examples of image/video fingerprinting are detailedin patent publications U.S. Pat. No. 7,020,304 (Digimarc), U.S. Pat. No.7,486,827 (Seiko-Epson), 20070253594 (Vobile), 20080317278 (Thomson),and 20020044659 (NEC). The disclosures of these patents are incorporatedherein by reference, in their entireties.

So, a displayed image or graphic, once optically captured, can triggerprovision of a user profile. But I want more. I want to communicatedifferent user profiles for different settings or experiences. Afterall, I might not want a business colleague to have access to my personaluser profile or friend user profile.

One way to accomplish transfer of different user profiles is throughconveying different orientation information. For example, orientationinformation can be conveyed through different positioning (e.g.,different orientations) of a device display screen when it isphotographed. I can hold the mobile device a first way (see FIG. 2A)when I want to transfer a first profile, hold the device a second way(see FIG. 2B) when I want to transfer a second profile, and hold thedevice a third way (see FIG. 2C) to facilitate the transfer of a thirdprofile, and so on.

Recall from above that a watermark may include a component from whichorientation information can be determined. So, for example, if mybackground or other displayed image/graphic includes a digital watermarkwith an orientation component, holding the displayed, watermarkedbackground at different orientations will introduce differentorientation information when the digital watermark is decoded.(Beforehand, I can associate different orientations or orientationranges with different profiles when I register a particular watermark orparticular watermarked background. For example, I can do this, e.g.,with an app running on my mobile device, or through an online portalthat manages my different profiles as discussed above.)

A tolerance can be included so that minor orientation variations duringimaging do not change my intended profile selection. For example, therotation in FIG. 2A is near 0 degrees. The first profile can beassociated within ±30 degrees of 0 degrees. The rotation in FIG. 2B isnear 90 degrees, and the second profile can be associated within ±30 of90 degrees. The rotation in FIG. 2C is near 270 degrees, and the thirdprofile can be associated within ±30degrees of 270 degrees. (Of course,0, 90 and 270 degrees, along with the ±30 degree range, are used by wayof example. Other orientations and ranges can be selected.)

Let's explore this further. Say I encounter a friend at while kickingthe soccer ball with my kids at the park. We haven't seen each other fora while so I want to update my friend my most recent digits (e.g., myup-to-date friend user profile). I've previously set my background (orother image) on my mobile phone to include a digital watermark, and I'veassociated my “friend profile” with a certain orientation (e.g., 0degrees, with some allowable tolerance). I hold my mobile device up andmy friend takes an image or video of my display screen, capturing mywatermarked background or other image. The watermark is decoded toobtain a payload, and the watermark orientation information isdetermined. Both the payload and the watermark orientation informationare conveyed to a remote database. There, the watermark and orientationinformation are used to find my “friend user profile,” which can beprovided to my friend's device, or made available for their access(e.g., stored in the cloud). In one implementation, the watermark can beused to access a database, record or table associated with me or mymobile device. The watermark orientation information can be then used tolocate a particular user profile within the database, record or table.

If I later encounter a potential business client, I can hold my phone ata different orientation before image capture. (I've previouslyassociated the different orientation (e.g., 90 degrees rotation) with my“business profile”.) This will allow transfer of my business profilethrough the same decoded watermark payload but with different watermarkorientation information.

While the above examples discuss rotation-based orientation thedisclosure is not so limited. Indeed, a display screen can be held atother orientations to include perspective, translation, scale, etc. Thisinformation can be used to further refine selection of a particular userprofile.

Instead of using different orientations, different images can be used toconvey or select different user profiles. For example, a first image canbe associated with a first user profile, a second image can beassociated with a second user profile, and so on. When encountering afriend, retailer or business client, I can quickly present theappropriate image and let them capture such with their mobile devicescamera. If I have a touchscreen, I can merely “swipe” the screen to findthe right picture. (An app can be configured to present images fordisplay, or my default background screen—probably through the deviceoperating system—can be configured to allow swiping to differentimages.) Each image may have a unique content identifier associatedtherewith (e.g., an encoded digital watermark or registeredfingerprint). Once decoded or derived, the content identifier can beforwarded to a database for identification, and the person can haveaccess to one or more of my particular user profiles.

My different images can have semantic value to help me remember whichimage should be displayed. For example, an image associated with myfriend user profile may depict me with a group of friends; or, an imageassociated with my business profile may depict my company logo or me allcleaned up in a suit and tie. My retailer profile might include adepiction of a bank or check, and so on.

Instead of using completely different pictures, I can use subtlevariations of the same picture, each with a different watermark payloadembedded therein or with a registered fingerprint of each. That way Ican conceal which profile I am providing someone. For example, a firstvariation might include a slightly cropped version of a base image, ormay include a few blacked out or altered pixels, or may include adifferent brightness or color variation. The different contentidentifiers can be associated with different profiles. This may helpprevent a third party from distinguishing which profile I am allowingaccess.

Another example includes placing a finger or thumb over the display. Thefinger or thumb is captured along with the displayed image orbackground. Pattern recognition can be employed to recognize the thumbor finger. This would not necessarily have any meaning to the persontaking the picture, but I could have this preprogrammed (via aregistration process) to help select a user profile. For example,perhaps I only cover the display with a thumb when presenting to aperson of dubious character. The recognized thumb is used to select a“stripped-down” user profile, one that only includes the barest ofpublic information (e.g., name and work phone).

In some cases, the mobile device can decide which profile should bepresented. For example, time, location and calendar information can beused to automatically determine which profile should be presented.Mobile devices often include a GPS or other location means. A mobiledevice could then use current location information from the GPS or otherlocation mean, perhaps in conjunction with an online service (e.g.,Google Maps), to determine which user profile to select. For example, ifthe current location information indicates that I'm in a Sporting Goodsstore the retailer profile can be selected (e.g., via a displayed imagehaving a content identifier associated with my retailer profile). Or, ifmy current location is in an office park, then my business profile canbe selected (e.g., via a displayed image having a content identifierassociated with my business profile). Still further, if I'm currently atthe pub, then perhaps my friends profile is selected by my mobile device(or an app running on the mobile device). In an alternative arrangement,if the location information provides ambiguous results, an app runningon the device can present two or more likely user profiles for myselection.

Time can also aid selection of an appropriate profile. For example, ifit's after 6 pm, my default profile may become my friend user profile.Or, an electronic calendar hosted online or on my device may beconsulted to see if I'm scheduled to be with friends, be at work,meeting new clients, etc. This information can be used to select a userprofile.

Some phones also include sensors (e.g., an accelerometer and microphone)which can provide relevant information. Different inferences can be madeif the sensor data meets certain criteria. For example, if a sensorindicates the device is traveling at 400 mph, it's likely that I'm on anairplane. A profile can be selected based on this information. Or, ifthe microphone is hearing certain types of noise or music (e.g., loudmusic may indicate that I'm busting moves at a night club, etc.) then aninference can be made to select a predetermined user profile. Noise andmusic characterization techniques have been used, e.g., to selectrelated music for purchase. For example, my music library may beanalyzed to determine other music of the same or related genres that Imight enjoy. Similar techniques, along with signal amplitude (e.g.,volume) can be used to analyze noise or sounds heard by a mobile deviceto help select a user profile.

Although much better than even 5 years ago, cell coverage (or wirelessaccess) isn't a guarantee. A mobile device can be programmed (e.g.,running an app) to store information when cell or network service isunavailable. Watermarks/fingerprints can be queued up and processed oncecell/wireless coverage is restored.

Various safeguards can be used to help protect against unauthorizedaccess to a user profile. For example, a user of a second mobile devicemay be tempted to store image data captured from a first mobile device'sdisplay screen. This user may then post the image data on a blog for allto use. Or an unscrupulous retailer may retain an image captured from acustomer's display screen in hopes of scoring multiple sales.

One safeguard utilizes the techniques disclosure in Digimarc's U.S. Pat.No. 7,111,168, which is incorporated herein by reference, in itsentirety. For example, a hash of captured imagery is obtained, passed tothe user profile repository for comparison with other image hashespreviously obtained. The presence of the hash in the database mayindicate an attack.

Another safeguard utilizes time information. For example, a displayedimage can include a time indicator graphic (2:37 PM, Sep. 15, 2011). SeeFIG. 3. The time indicator is captured along with the image. OpticalCharacter Recognition (OCR) software can be used to recognize the time.This information can be used to limit access to a user profile. Forexample, if the time indicator is not within 1 minute (or 15 seconds, or15 minutes, etc.) of a request for a user profile, then access isdenied. A watermark can also include a time/date stamp. The time/datestamp can be used to determine whether the request is fresh enough toallow access, or for comparison against the time indicator graphic.

Digital signatures can also be used as a safeguard. Additionally,traditional challenge and response techniques (e.g., akin to one-timepads) can also be used to help safeguard such exchanges. A digitalwatermarking embedding process may be altered based upon somepredetermined timing arrangement. This altered embedding process can beused each time an image is to be displayed on the screen, introducing anew watermark (or payload) for each user profile exchange. The mobiledevice and the online user profile repository can be synced so that theycorrespond. The watermark will only be good for a predetermined time, orfor a limited number of user profile access requests.

The present disclosure is not limited to capturing images displayed on amobile device's display screen. For example, instead of displaying adigital still image, a first mobile device could render a video clip. Asecond mobile device could capture imagery from the video clip anddecode or derive a content identifier from such. That content identifiercould then be used to obtain a user profile associated with a user ofthe first mobile device.

Likewise, audio could be used to transfer different user profiles. Afirst audio clip (or first audio associated with video) may beassociated with a first user profile, a second audio clip (or secondaudio associated with video) could be associated with a second userprofile, and so on. Thus, the present disclosure is not limited tocontent identifier obtained from still imagery, and the methods andapparatus discussed above could be used with video and/or audio as well.

To provide a comprehensive disclosure without unduly lengthening thisspecification, applicants incorporate-by-reference the documents andpatent disclosures referenced above. (Such documents are incorporated intheir entireties, even if cited above in connection with specific oftheir teachings.) These references disclose technologies and teachingsthat can be incorporated into the arrangements detailed herein, and intowhich the technologies and teachings detailed herein can beincorporated.

What is claimed is:
 1. A method employing a mobile device having aprocessor configured to perform one or more acts of the method, themobile device including at least one sensor, said method comprising:obtaining information from the sensor; selecting a user profile fromamong a plurality of different user profiles based on the information.2. The method of claim 1 in which the sensor comprises an accelerometerfor indicating relative movement.
 3. The method of claim 2 in which theinformation comprises relative movement.
 4. The method of claim 1 inwhich the sensor comprises a microphone for capturing ambient audio. 5.The method of claim 4 in which the information is inferred based oncaptured ambient audio.
 6. The method of claim 5 in which theinformation is inferred based on an audio noise level of capturedambient audio.
 7. The method of claim 5 in which the information isinferred based on signal characteristics of captured ambient audio. 8.The method of claim 1 in which the sensor comprises a global positioningsystem receiver (GPS receiver).
 9. The method of claim 8 in which theinformation comprises location information.
 10. The method of claim 1 inwhich the plurality of different user profiles comprises a social mediaprofile and a retail profile.
 11. The method of claim 1 in which theplurality of different user profiles is associated with differentinformation in memory or a database.
 12. The method of claim 1 in whichthe mobile device further comprises a display screen and said methodfurther comprises selecting—based on a selected user profile—an image orgraphic for display on the display screen, the image or graphic beingassociated with the user profile.
 13. The method of claim 12 furthercomprising embedding a digital watermark in a selected image or graphic,the digital watermark including a plural-bit payload associated with theselected user profile.
 14. A method comprising: populating a pluralityof different user profiles; obscuring which user profile out of theplurality of different user profiles is selected by selection of animage from a plurality of different images; displaying a selected imageon a display screen of a first mobile device for image capture by asecond mobile device.
 15. The method of claim 14 in which each image inthe plurality of different images comprises a variation of one another.16. A method employing a mobile device having a processor configured toperform one or more acts of the method, the mobile device including orcommunicating with a clock: obtaining information from the clock; andselecting a user profile from among a plurality of different userprofiles based on the information.
 17. An electronic device programmedto perform the method of claim
 16. 18. A non-transitory computerreadable medium comprising instructions stored thereon to cause aprocessor to perform the method of claim
 16. 19. A mobile devicecomprising: a sensor; and a processor programmed for: obtaininginformation from the sensor; selecting a user profile from among aplurality of different user profiles based on the information.
 20. Themobile device of claim 19 in which the sensor comprises an accelerometerfor indicating relative movement.
 21. The mobile device of claim 20 inwhich the information comprises relative movement.
 22. The mobile deviceof claim 19 in which the sensor comprises a microphone for capturingambient audio.
 23. The mobile device of claim 22 in which theinformation is inferred based on captured ambient audio.
 24. The mobiledevice of claim 22 in which the information is inferred based on anaudio noise level of captured ambient audio.
 25. The mobile device ofclaim 22 in which the information is inferred based on signalcharacteristics of captured ambient audio.
 26. The method of claim 19 inwhich the sensor comprises a global positioning system receiver (GPSreceiver).
 27. The mobile device of claim 26 in which the informationcomprises location information.
 28. The mobile device of claim 19further comprising memory, in which the plurality of different userprofiles is associated with different information in said memory. 29.The mobile device of claim 19 further comprising a display screen, inwhich said processor is programmed for selecting—based on a selecteduser profile—an image or graphic for display on the display screen, theimage or graphic being associated with the user profile.
 30. The mobiledevice of claim 29 in which said processor is programmed for embedding adigital watermark in a selected image or graphic, the digital watermarkcomprising a plural-bit payload associated with the selected userprofile.